Rethinking the Agreement in Human Evaluation Tasks

Human evaluations are broadly thought to be more valuable the higher the inter-annotator agreement. In this paper we examine this idea. We will describe our experiments and analysis within the area of Automatic Question Generation. Our experiments show how annotators diverge in language annotation tasks due to a range of ineliminable factors. For this reason, we believe that annotation schemes for natural language generation tasks that are aimed at evaluating language quality need to be treated with great care. In particular, an unchecked focus on reduction of disagreement among annotators runs the danger of creating generation goals that reward output that is more distant from, rather than closer to, natural human-like language. We conclude the paper by suggesting a new approach to the use of the agreement metrics in natural language generation evaluation tasks

Evaluation methodologies in Automatic Question Generation 2013-2018

In the last few years Automatic Question Generation (AQG) has attracted increasing interest. In this paper we survey the evaluation methodologies used in AQG. Based on a sample of 37 papers, our research shows that the systems’ development has not been accompanied by similar developments in the methodologies used for the systems’ evaluation. Indeed, in the papers we examine here, we find a wide variety of both intrinsic and extrinsic evaluation methodologies. Such diverse evaluation practices make it difficult to reliably compare the quality of different generation systems. Our study suggests that, given the rapidly increasing level of research in the area, a common framework is urgently needed to compare the performance of AQG systems and NLG systems more generally

An overview and analysis of community bank mergers

With some of the largest mergers in history now taking place in the financial services industry, the fact that consolidation is also occurring among small banking institutions is often overlooked. The factors that are promoting consolidation in the banking industry are also relevant for the smallest banks, namely the need to spread the cost of technological and administrative overhead and the desire to maintain earnings growth. With limited growth opportunities in many rural communities, smaller banks often choose to merge with other nearby rural banks as the means to gain asset size and improve efficiency. ; Using a case study approach that focuses on nineteen rural banks that participated in in-market mergers, this article examines whether smaller community banks that followed this merger strategy realized efficiency gains. The results show that such mergers have usually been successful from both a profitability and a cost efficiency perspective. Further, these gains were typically achieved without closing branch offices. These successes are important to rural bankers as they seek opportunities for consolidation. They are also important from a public policy perspective and should be carefully considered by regulators in their evaluation of small bank mergers.Bank mergers

Using discovered, polyphonic patterns to filter computer-generated music

A metric for evaluating the creativity of a music-generating system is presented, the objective being to generate mazurka-style music that inherits salient patterns from an original excerpt by Frédéric Chopin. The metric acts as a filter within our overall system, causing rejection of generated passages that do not inherit salient patterns, until a generated passage survives. Over fifty iterations, the mean number of generations required until survival was 12.7, with standard deviation 13.2. In the interests of clarity and replicability, the system is described with reference to specific excerpts of music. Four concepts–Markov modelling for generation, pattern discovery, pattern quantification, and statistical testing–are presented quite distinctly, so that the reader might adopt (or ignore) each concept as they wish

Letter to Leah Chanin regarding SEAALL Constitution and Bylaws, June 16, 1972

A letter from Paul Willis to Leah Chanin enclosing drafts of the revised SEAALL Constitution and Bylaws

Letter to Pearl Von Allmen regarding SEAALL Constitution and Bylaws, June 16, 1972

A letter from Paul Willis to Pearl Von Allmen enclosing drafts of the revised SEAALL Constitution and Bylaws

Letter to Pearl Von Allmen regarding the Dorothy Salmon Memorial Fund, April 9, 1969

A letter from Paul Willis to Pearl Von Allmen thanking SEAALL for a contribution to the Dorothy Salmon Memorial Fund

Note to Pearl Von Allmen regarding SEAALL Constitution and Bylaws, September 25, 1972

A note from Paul Willis to Pearl Von Allmen providing additional changes to the SEAALL Constitution and Bylaws